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SUMMARY 

An  assessment  of  a fully  operational  command 
centre  performing  under  high  pressures  in 
information  load,  time  stress  and  cognitive 
complexity  has  shown  that  in  particularly,  four 
factors  play  a crucial  role:  individual  information 
processing,  team  management,  communication  load 
and  the  distribution  of  tasks.  It  is  concluded  that 
individual  processes  and  team  processes  compete 
with  each  other.  If  individuals  are  getting  loaded 
then  first  those  tasks  will  be  dropped  that  are 
demanding  and  do  not  lead  towards  direct  feedback. 
Team  tasks  suffer  most  under  these  conditions. 
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1 INTRODUCTION 

Recently,  the  Royal  Netherlands  Navy  has  launched 
several  studies  for  the  analysis  and  design  of  the 
operational  effectiveness  of  command  crew.  These 
aim  at  understanding  better  the  critical 

requirements  of  command  centres.  An  important 
driver  of  these  studies  is  the  need  to  optimise  crew 
size  and  operational  performance.  In  this 
presentation  I will  focus  on  the  assessment  of  the 
current  M-frigate  class  command  centre  that  was 
performed  in  1 998  by  a team  of  researchers  of  TNO 
Human  Factors. 

The  attention  towards  command  crew  effectiveness 
is  one  that  fits  a gradual  development  of  human 
factors  studies  for  the  Dutch  Navy.  These  had  an 
early  focus  on  workplace  ergonomics,  and 
developed  gradually  into  interface  design  and 
common  visual  spaces,  into  assessment  of  the 
human-machine  complex  in  the  current  command 
arrangement  and  for  future  command  centres.  In 
parallel,  efforts  are  being  employed  to  structurally 


incorporate  human  factors  in  the  broadest  sense  into 
the  conceptual  development  and  the  design  process 
(Human  Systems  Integration,  US  DOD  5000.1, 
1992;  UK  Human  Factors  Integration  Program, 
D/DOR  (Sea),  1991;  Beevis,  D.,  Essens,  P.  & 
Schuffel,  H.,  1996). 

Command  centres 

A command  centre  (operations  room,  or  combat 
information  centre)  of  a frigate  is  a typical  example 
of  a complex  system  where  information  from 
different  ‘worlds’  (air,  surface  and  subsurface)  is 
gathered,  analysed  and  acted  upon. 

A command  centre  can  be  characterised  as  a 
'human  activity  system'  (Checkland,  1993)  designed 
for  human  information  processing,  decision  making 
and  execution  supported  by  technologies  such  as 
interfaces  to  sensor  and  weapon  systems,  and 
combat  information  system.  The  system  is  built  to 
respond  to  a wide  variety  of  situations  and  signals 
from  external  worlds,  that  are  dynamic  and  change 
with  varying  time  horizons.  The  system  is  complex 
because  it  is  comprised  of  multiple  people  and 
multiple  technologies  organised  in  several 
subsystems.  These  have  to  tune  their  processes  and 
combine  and  collate  the  information  produced  in 
order  to  achieve  the  mission  of  the  ship. 

Traditional  human  factors  evaluations  of  complex 
prototypes  or  operational  systems  usually  focus  on 
operator  performance  instead  of  performance  of  the 
whole  system  (Meister,  1998),  or  limits  evaluation 
to  a sub-system,  such  as  the  human-computer 
interface,  or  to  a single  criterion,  such  as  workload. 
Outcome  measures  of  the  whole  system  may  be 
used  to  get  an  summative  view  on  the  performance 
of  the  whole  system  (often  referred  to  as  MOEs). 
The  problem  is  however  that  a systems  outcome 
approach  treats  the  system  as  'black  box'  (input- 
output)  hiding  multiple  internal  sources  of  positive 
and  negative  effects  which  lead  to  some  composite 
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result.  Therefore  these  measures  fail  to  provide  an 
understanding  of  the  factors  that  generate  that 
outcome.  The  fundamental  problem  however  is  that 
there  is  still  insufficient  knowledge  of  the 
contribution  of  one  factor  to  the  outcome  of  the 
whole  system  or  what  factors  contribute  most. 

2 M-FRIGATE  COMMAND  CENTRE 

The  study  of  the  current  command  arrangements  in 
the  M-frigate  - the  focus  of  this  paper  - was  driven 
by  the  Navy’s  need  to  understand  better  which 
factors  are  critical  for  effective  performance  in 
order  to  optimise  current  command  and  to 
incorporate  those  in  the  design  of  future  ship  and 
command  centres. 

A command  centre  in  full  operation  usually 
displays  an  impressive  hectic  and  concentrated 
activity  often  with  intense  verbal  communications 
via  networks  and  face-to-face.  Assessment  of 
performance  under  these  conditions  requires  a well 
developed  methodology  and  instruments  that 
capture  the  dynamics  of  the  processes  and 
addresses  the  critical  factors  of  the  complex  system. 
Our  tasks  was  to  develop  such  methodology. 

Of  other  studies  in  Naval  command,  in  particular 
the  TADMUS  studies  should  be  mentioned 
(Cannon-Bowers  & Salas,  1998).  Similarly  to  goals 
in  our  command  studies  they  study  the  complex 
nature  of  command  in  dynamic  and  stressful 
situations.  A difference  is  that  they  have  focused  on 
the  air  defence  team,  while  we  look  at  the 
command  centre  as  a whole  representing  the 
mission  of  the  ship. 

Moreover,  the  M-frigate  assessment  was  directed 
specifically  to  the  factors  that  determine 
effectiveness  of  the  current  command  centre.  This 
will  then  be  used  as  a baseline  for  further  studies. 
The  assessment  does  not  address  the  strategic  or 
tactical  outcome  of  decision  making,  nor  the 
performance  of  the  technologies  used,  rather  it 
focuses  on  individual  and  team  processes  in 
information  processing  and  decision  making. 

Performance  indicators 

High  workload  conditions  are  considered  to  be 
‘normal’  in  an  operational  command  centre.  The 
question  is  whether  workload  levels  are  that  high 
that  processing  capacity,  of  individual  or  team, 
shows  its  limits.  This  would  negatively  affect 
performance  effectiveness  - there  is  more  to  be 
done  than  can  be  handled  in  the  time  given.  A 
second  question  is  whether  workload  is  evenly 


distributed  over  the  crew.  If  in  a critical  situation 
some  team  members  have  not  much  at  hand  while 
others  are  overloaded,  then  resources  are  apparently 
under-utilised.  But,  not  only  from  an  operational 
perspective  is  workload  an  important  issue.  Also 
for  the  crew’s  appreciation  and  quality  of  work: 
constant  performing  below  expectations  will  result 
in  negative  feelings  and  consequently  in  lower 
performance. 

In  dynamic  event-rich  situations  measuring 
workload  once  at  the  end  of  a mission  would  be 
insufficient.  A continuous  review  of  the  workload 
gives  insight  how  workload  develops  over  time  for 
each  crew  member,  where  bottlenecks  are  in 
processing  capacity,  and  how  workload  is 
distributed  over  the  crew  members. 

Workload  can  be  used  as  an  indicator  of  efficient 
and  effective  team  performance  but  does  not  show 
how  well  the  job  is  done.  Qualification  of  how  well 
a task  is  being  performed  requires  insight  in  how 
the  demands  of  an  operational  situation  are 
addressed.  In  complex  tasks  one  should  also  take 
into  account  that  there  are  parallel  tasks  to  be 
processed  and  prioritisation  or  effective  task 
switching  can  only  be  judged  in  context.  This  is  the 
role  of  the  domain  expert.  Judgement  of  quality 
should  be  coupled  with  critical  events  in  the 
scenario's  which  makes  it  easier  to  score  during  a 
sessions. 

METHOD 

Experimental  setting 

Obviously,  in  order  to  assess  the  effectiveness  of  a 
command  centre  under  critical  conditions  one  needs 
a realistic  setting,  with  realistic  scenario’s,  with  a 
team  that  is  fully  operational. 

On  the  other  hand,  for  producing  well-founded 
conclusions,  one  needs  control  over  the  situation, 
repeated  measurement  using  multiple  teams 
responding  to  the  same  situations.  Moreover,  given 
time  and  resources  available  this  has  to  be  done  in 
an  efficient  way. 

We  have  used  the  Navy’s  training  facility  which 
has  a copy  of  the  M-frigate  command  centre.  The 
use  of  matching  equipment  and  layout  facilitates 
the  application  of  skill-based  activities  so  that  the 
teams  can  perform  the  way  they  are  used  to;  only 
the  variations  in  the  scenario’s  will  show  in  the 
performance. 
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The  use  of  a land-based  facilities  always  evokes 
discussions  of  the  effect  of  ship  movements  on  the 
performance.  Indeed,  ship  movements  result  in 
fatigue  which  affects  performance.  Moreover,  there 
may  be  effects  on  team  tasks  such  as 
communication  and  team  management.  A second 
aspect  of  reality  not  easily  simulated  in  a training 
facility  (and  in  standard  sea-based  exercises)  is  the 
stress  of  being  under  attack.  It  was  argued  that 
these  factors  would  certainly  add  to  performance 
degradation,  but  that  first  stress  due  to  time-limits 
and  information-uncertainty  should  be  investigated. 
In  order  to  augment  the  validity  of  the  assessment, 
the  findings  from  the  experimental  situations  were 
verified  during  full  sea  exercises. 

Registration  of  workload 

All  workstations  were  equipped  with  a special 
keypad  with  five  keys  representing  a 5 -point  scale: 
Most  left  key  represents  score  1 defined  as  ‘time 
left’;  most  right  key  defined  as  score  5 indicating 
‘too  little  time’.  The  middle  key  (3)  was  defined  as 
‘just  enough  time’.  The  operator’s  task  was  to 
indicate  how  much  time  was  left  for  doing  their 
jobs  at  that  very  moment.  Every  five  minutes  a 
small  lamp  on  the  keypad  flashed  which  cued  the 
team  member  to  give  a judgement.  Each  key  press 
was  registered  in  a computer-system  and  linked  to 
video  registration  (Observer  system). 

Registration  of  performance  quality 

Four  indicator  categories  are  distinguished:  the 
quality  of  the  information  processing  of  individual 
crew  members,  the  communications  between  the 
team  members,  the  handling  and  application  of  the 
systems  and  tools,  and  the  management  of  team  by 
the  team  leader.  The  categories  cover  several 
subcategories: 


‘Information  processing’  comprises  delay  in 
information  processing,  overlooking  of 
information,  errors  in  interpretation; 
‘Communications’  concerns  the  correct 
application  of  communications  procedures, 
timely  communications,  interrupting 
communications; 

‘System  handling’  covers  the  ease  of  operating 
a system,  lack  of  skills  in  operation;  adequacy 
of  settings; 

‘Team  management’  refers  to  limitations, 
adequacy  or  overdoing  in  direction  and  control. 
Seven  domain-experts  were  distributed  over  the 
team  and  observed  performance.  When  they  judged 
during  the  session  that  one  or  multiple  categories 
were  applicable  in  that  situation  this  was  scored  on 
a form  and  with  a key  press  the  instance  and  its 
weight  was  linked  to  the  video  recording. 

Scenario’s 

Scenario’s  are  sequences  of  predefined  conditions 
and  events.  Assessment  of  a system  requires  critical 
events  of  combination  of  events  that  load  of  even 
test  the  limits  of  the  central  functions  of  a system. 
For  a command  centre  central  functions  are 
processing  information  and  making  decisions. 
Three  scenario’s  were  developed:  a normal 
workload  scenario’s  of  which  was  expected  that  a 
well-trained  team  could  handle  without  problems; 
an  input  information-load  scenario  with  many 
tracks  and  information  units;  and,  an  information- 
uncertainty  scenario  with  missing  information  and 
unexpected  behaviour  of  the  objects  in  the  world. 

Scenario  design  was  based  on  our  knowledge  of 
factors  that  affect  information  processing  and 
decision  making,  such  as: 

number  of  tracks  and  multiple  threats  or  targets, 
and  number  of  potential  measures  relate  to 
overload  of  information  handling; 
conflicting  information,  ambiguous  cues, 
threats,  multiple  options  and  criteria, 
conflicting  use  of  resources  relate  to  cognitive 
stress  and  processing  capacity; 
insufficient  information,  misleading  or  novel 
situations,  limited  feedback  relate  to 
availability  of  knowledge  and  experience. 

Procedure 

Four  experienced  and  fully-operational  teams  of  16 
members  each  participated  in  the  assessment 
sessions  performing  in  three  scenario’s.  At  the  start 
of  the  session  they  received  a standard  operational 
briefing  which  explained  the  operational  situation 
and  mission.  Also  was  explained  that  measurement 
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tools  were  used  to  get  an  indication  of  the 
functioning  of  the  command  centre  and  that  no 
individual  scores  were  isolated.  The  scenario 
sequence  was  the  same  for  each  team:  normal, 
information-load,  information-uncertainty  scenario. 
Each  scenario  took  about  40  minutes.  Between  each 
scenario  was  a break.  After  the  session  a 
questionnaire  was  handed  out  which  addressed  the 
functioning  of  the  command  centre  on  board. 
Subsequently,  in  a debrief  scenario  events  and 
typical  performance,  actions  and  decisions  were 
discussed  by  the  team. 

RESULTS 

Workload  data  showed  that  about  half  of  the  scores 
are  around  the  middle  level  (‘just  time  enough’) 
which  can  be  considered  as  the  standard  level  of 
load  during  operations.  One  third  of  the  scores  was 
below  that  level  indicating  that  there  was  ample 
time  for  performing  the  tasks  at  hand.  One  sixth  of 
the  scores  are  above  the  just  enough  level 
indicating  that  there  were  instances  that  there  was 
very  little  time  to  perform  the  task  at  hand.  Overall 
levels  of  the  scores  can  be  clustered  around 
subgroups  in  the  command  centre.  It  was  found  that 
the  command  team  has  on  average  a higher 
workload  then  the  supporting  operators,  with 
exception  of  a particular  subgroup  of  operators  (the 
electronic  warfare  operator  and  controller)  who  had 
also  a high  workload.  There  was  little  difference 
between  the  two  test  scenario’s.  The  conclusion  is 
that  the  unequal  distribution  of  workload  found 
results  from  an  organisation  of  tasks  which,  under 
complex  conditions,  centralises  control  with  the 
command  team. 
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Quality  assessment  of  performance  showed  several 
opportunities  for  improvement.  Of  most  interest  is 
the  distribution  of  the  scores  over  the  categories. 
The  domain  experts  particularly  judged  that  team 
direction  and  control  (35%)  and  individual 


information  processing  (30%)  could  be  improved; 
less  prominent  were  system  handling  (18%)  and 
communications  (17%). 

The  questionnaire  showed  that  internal 
communications  was  considered  to  require  a high 
level  of  attention  (90%  of  respondents,  N=53),  and 
that  its  volume  was  considered  to  be  too  high 
(60%).  The  tasks  to  be  performed  are  in  general 
time  critical  and  require  high  attention  (88%). 
System  handling  could  be  improved  by  improving 
the  interface  with  more  direct  interaction.  In  time 
critical  situations  any  system  delay  is  considered 
hindering  performance. 

DISCUSSION 

The  assessment  of  the  command  centre  performing 
under  high  pressures  in  information  load  , time 
stress  and  cognitive  complexity  shows  that  in 
particularly,  four  factors  play  a crucial  role: 
individual  infonnation  processing,  team 
management,  communication  load  and  the 
distribution  of  tasks.  An  interesting  statement  from 
a Naval  evaluator  describes  the  combination  of 
these  factors  most  adequately:  ‘when  load  increases 
in  the  team  curtains  seem  to  get  shut  between  team 
members’. 

In  trying  to  understand  the  combination  of  factors 
we  distinguished  between  the  structural  settings  of 
the  command  centre  and  the  command  and  control 
process  itself.  Structural  is  the  given  quality  of  the 
team  and  their  ability  to  work  together.  Structural  is 
the  distribution  of  tasks  and  the  work  procedures 
that  are  predefined.  Structural  is  also  the 
technology  used  to  support  or  enable  the  command 
centre  to  do  its  job.  These  factors  are  brought  into 
play  before  the  mission  starts. 

During  the  operations  people  are  the  flexible  and 
vulnerable  force  and  adopt  compensation  strategies 
for  what  is  structurally  imperfect:  a situation  most 
people  recognise  immediately.  From  the  literature 
on  stress  handling  a checklist  of  strategies  was 
derived  (Gaillard,  1 996): 

first  people  try  to  compensate  for  situations  the 
system  was  not  designed  for  by  putting  extra 
effort  into  play 

if  load  rises  further  tasks  for  which  that  is 
possible  are  delayed  or  postponed 
subsequently,  focus  is  on  a limited  set  of  tasks; 
at  best  the  central  and  most  critical  tasks 
less  communications  is  observed  with  an 
increasing  tolerance  for  errors  (correction  costs 
time  and  effort) 
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finally,  control  over  other  people  is  loosened 
and  eventually  lost. 

This  list  shows  the  process  that  can  also  be 
observed  in  highly  loaded  conditions  in  command 
centres.  It  shows  that  individual  factors  and  team 
factors  interact  with  each  other  in  an 
understandable  way.  If  individuals  are  getting 
loaded  and  then  first  those  tasks  will  be  dropped 
that  are  demanding  and  do  not  lead  towards  direct 
success  or  hinder  their  most  direct  responsibilities. 
Team  tasks  can  be  considered  to  be  such  tasks.  In 
particular  when  team  leaders  have  also  other  tasks 
than  direction,  control  and  co-ordination  the  danger 
is  that  they  will  do  the  direct  accountable  tasks  first 
and  postpone  other  tasks  to  later.  Eventually  this 
may  result  in  a breaking  down  of  the  team  as  a 
whole. 

In  conclusion,  the  following  factors  are  considered 
critical  for  effectiveness: 

- Organisation  of  processing  capacity:  focussed  on 
optimised  task  distribution;  flexibility  in  task 
allocation;  reduced  communications 

- Quality  of  people  / team:  focussed  on  training  as 
a team;  training  graceful  degradation  in  order  to 
deal  with  overload  situations;  training  in  complex 
and  varied  situations 

- Technology:  focussed  on  directness  in  interaction 
and  response;  support  in  handling  information. 
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